Add system(command; args) operator (disabled by default) by Copilot · Pull Request #2640 · mikefarah/yq

Copilot · 2026-03-27T10:03:57Z

Adds a system operator that executes an external command and returns its stdout as a string value. Disabled by default for security — requires explicit opt-in via --security-enable-system-operator.

Behaviour

Disabled (default): logs a warning, returns null for each matched node
Enabled: runs the command, pipes the current node's YAML representation to stdin, returns stdout with one trailing newline stripped as a !!str

# Without flag: warns and returns null
yq '.country = system("/usr/bin/echo"; "test")' data.yml
# => country: null

# With flag: executes the command
yq --security-enable-system-operator '.country = system("/usr/bin/echo"; "test")' data.yml
# => country: test

# Array arguments
yq --security-enable-system-operator '.value = system("/bin/echo"; ["hello", "world"])' data.yml
# => value: hello world

# No arguments (stdin-only)
yq --security-enable-system-operator '.result = system("/usr/local/bin/my-filter")' data.yml

Changes

security_prefs.go — adds EnableSystemOps bool (default false)
operator_system.go — operator implementation; supports system("cmd"), system("cmd"; "arg"), and system("cmd"; ["arg1", "arg2"]); captures stderr in error messages; validates command is a non-null, non-empty scalar; evaluates command/args per matched node using SingleReadonlyChildContext
operation.go — registers systemOpType
lexer_participle.go — registers simpleOp("system", systemOpType)
cmd/root.go — adds --security-enable-system-operator persistent flag (consistent with --security-disable-env-ops / --security-disable-file-ops)
operator_system_test.go — covers disabled/enabled states, array args, per-node expression evaluation, null command error, and command failure; uses exec.LookPath for portability
doc/operators/headers/system-operators.md — documentation header

📱 Kick off Copilot coding agent tasks wherever you are with GitHub Mobile, available on iOS and Android.

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/8a11e9a0-10d2-4f2a-ae29-4e9d0bfc266f Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

Copilot

Pull request overview

Adds a new system(...) operator to yqlib that can execute external commands (disabled by default) and wires it through parsing, CLI configuration, tests, and operator documentation.

Changes:

Introduces SecurityPreferences.EnableSystemOps (default false) and a --enable-system-operator CLI flag to opt in.
Implements system operator parsing/registration and execution logic (stdin = current node YAML, stdout -> string).
Adds operator scenarios + generated documentation pages/headers.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
pkg/yqlib/security_prefs.go	Adds security preference toggle for system operator.
pkg/yqlib/operator_system.go	Implements `system` operator execution and result mapping.
pkg/yqlib/operation.go	Registers `SYSTEM` operation type.
pkg/yqlib/lexer_participle.go	Adds `system` to the lexer/operator table.
cmd/root.go	Adds `--enable-system-operator` persistent flag.
pkg/yqlib/operator_system_test.go	Adds scenarios for disabled/enabled behavior and failures.
pkg/yqlib/doc/operators/system-operators.md	Adds generated operator docs with examples/output.
pkg/yqlib/doc/operators/headers/system-operators.md	Adds doc header snippet included by generator.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-27T11:14:41Z

pkg/yqlib/operator_system.go

+	var command string
+	var argsExpression *ExpressionNode
+
+	// check if it's a block operator (command; args) or just (command)
+	if expressionNode.RHS.Operation.OperationType == blockOpType {


command/argsExpression are resolved once using context.ReadOnlyClone() before iterating over context.MatchingNodes, and only the first match is used. This means system(<expr>) and system(<expr>; <expr>) cannot vary per matched node (e.g. system(.cmd; .args)), and may pick an unrelated first result when multiple nodes are matched. Consider evaluating the command/args expressions inside the per-candidate loop using context.SingleReadonlyChildContext(candidate) (and failing if they don't yield a single scalar/sequence) so parameter expressions behave consistently with other operators like load.

@copilot apply changes based on this feedback

Done in da611f7. Command and args expressions are now evaluated inside the per-candidate loop using context.SingleReadonlyChildContext(candidate), matching the pattern used by loadOperator. This allows expressions like system(.cmd; .arg) to resolve .cmd and .arg against each matched node individually. Also added a test case for per-node evaluation, used exec.LookPath for portability, and fixed CRLF-aware newline trimming.

pkg/yqlib/operator_system.go

Copilot · 2026-03-27T11:14:42Z

pkg/yqlib/operator_system_test.go

+		subdescription: "Use `--enable-system-operator` to enable the system operator.",
+		document:       "country: Australia",
+		expression:     `.country = system("/usr/bin/echo"; "test")`,
+		expected: []string{
+			"D0, P[], (!!map)::country: null\n",
+		},
+	},
+}
+
+var systemOperatorEnabledScenarios = []expressionScenario{
+	{
+		description:    "Run a command with an argument",
+		subdescription: "Use `--enable-system-operator` to enable the system operator.",
+		document:       "country: Australia",
+		expression:     `.country = system("/usr/bin/echo"; "test")`,
+		expected: []string{


These tests depend on hard-coded absolute paths like /usr/bin/echo and /bin/echo, which can fail on some Linux distros/containers and all non-POSIX platforms. Consider resolving the executable at runtime (e.g. via exec.LookPath) or skipping these scenarios when the required commands aren't available, to keep the test suite portable and reliable.

pkg/yqlib/operator_system_test.go

pkg/yqlib/doc/operators/system-operators.md

Copilot · 2026-03-27T11:14:42Z

pkg/yqlib/security_prefs.go

 type SecurityPreferences struct {
-	DisableEnvOps  bool
-	DisableFileOps bool
+	DisableEnvOps   bool
+	DisableFileOps  bool
+	EnableSystemOps bool
 }


SecurityPreferences currently uses a Disable* naming pattern (DisableEnvOps, DisableFileOps), but this introduces EnableSystemOps. For consistency and to reduce confusion when adding future prefs, consider renaming so all fields use the same polarity/pattern (e.g. DisableSystemOps with an inverted default, or rename the existing fields).

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…ldContext Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/dca841eb-3f63-4f23-adeb-556431560420 Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

…m op docs Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/3f8a5375-25fd-4428-a8e6-b630194c36b2 Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

pkg/yqlib/operator_system.go

Copilot · 2026-03-28T03:15:21Z

pkg/yqlib/operator_system.go

+			if commandNodes.MatchingNodes.Front() == nil {
+				return Context{}, fmt.Errorf("system operator: command expression returned no results")
+			}
+			command = commandNodes.MatchingNodes.Front().Value.(*CandidateNode).Value
+
+			argsNodes, err := d.GetMatchingNodes(nodeContext, block.RHS)
+			if err != nil {
+				return Context{}, err
+			}
+			if argsNodes.MatchingNodes.Front() != nil {
+				args = resolveSystemArgs(argsNodes.MatchingNodes.Front().Value.(*CandidateNode))
+			}
+		} else {
+			commandNodes, err := d.GetMatchingNodes(nodeContext, expressionNode.RHS)
+			if err != nil {
+				return Context{}, err
+			}
+			if commandNodes.MatchingNodes.Front() == nil {
+				return Context{}, fmt.Errorf("system operator: command expression returned no results")
+			}
+			command = commandNodes.MatchingNodes.Front().Value.(*CandidateNode).Value
+		}


command is taken from the first matched node’s .Value without validating type/tag. If the command expression yields !!null or a non-scalar node, exec.Command will be invoked with an empty/invalid command (leading to confusing errors like “exec: no command”). Consider validating that the command node is a non-null scalar (and possibly erroring if multiple results are returned) and returning a clearer system operator: command must be a string scalar error.

@copilot apply changes based on this feedback If there are multiple results, just use the first (and log a debug, not a warning, of all the matches).

Done in 53abbba. Extracted a resolveCommandNode helper that validates the command is a non-null scalar (returning "system operator: command must be a string scalar" if not) and logs a Debugf when multiple results are returned before using the first.

pkg/yqlib/doc/operators/headers/system-operators.md

pkg/yqlib/doc/operators/system-operators.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/928aabc5-ad71-41d8-94ab-403942e3f92d Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-28T09:42:42Z

pkg/yqlib/operator_system.go

+		result := string(output)
+		if strings.HasSuffix(result, "\r\n") {
+			result = result[:len(result)-2]
+		} else if strings.HasSuffix(result, "\n") {
+			result = result[:len(result)-1]
+		}


The PR description says stdout is "trimmed", but the implementation only strips a single trailing \n/\r\n. If the intent is to trim all trailing whitespace/newlines, use strings.TrimRight/TrimSpace; otherwise update the description/docs to match the current behavior ("strip one trailing newline").

Copilot · 2026-03-28T09:42:42Z

pkg/yqlib/operator_system_test.go

+func TestSystemOperatorEnabledScenarios(t *testing.T) {
+	echoPath := findExec(t, "echo")
+	falsePath := findExec(t, "false")
+
+	originalEnableSystemOps := ConfiguredSecurityPreferences.EnableSystemOps
+	defer func() {
+		ConfiguredSecurityPreferences.EnableSystemOps = originalEnableSystemOps
+	}()
+
+	ConfiguredSecurityPreferences.EnableSystemOps = true
+
+	scenarios := []expressionScenario{
+		{
+			description:    "Run a command with an argument",
+			subdescription: "Use `--enable-system-operator` to enable the system operator.",
+			yqFlags:        "--enable-system-operator",
+			document:       "country: Australia",
+			expression:     `.country = system("` + echoPath + `"; "test")`,
+			expected: []string{


The doc-generating enabled scenarios embed echoPath from exec.LookPath, which can vary across environments (e.g. /usr/bin/echo vs /bin/echo) and cause non-deterministic generated docs. Consider hardcoding a stable command path/name for scenarios that are included in docs (or marking these scenarios skipDoc: true and adding a separate deterministic doc scenario).

cmd/root.go

pkg/yqlib/operator_system.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

mikefarah · 2026-04-06T08:33:04Z

cmd/root.go

 	rootCmd.PersistentFlags().BoolVarP(&yqlib.ConfiguredSecurityPreferences.DisableEnvOps, "security-disable-env-ops", "", false, "Disable env related operations.")
 	rootCmd.PersistentFlags().BoolVarP(&yqlib.ConfiguredSecurityPreferences.DisableFileOps, "security-disable-file-ops", "", false, "Disable file related operations (e.g. load)")
+	rootCmd.PersistentFlags().BoolVarP(&yqlib.ConfiguredSecurityPreferences.EnableSystemOps, "security-enable-system-operator", "", false, "Enable system operator to allow execution of external commands.")
+	rootCmd.PersistentFlags().BoolVarP(&yqlib.ConfiguredSecurityPreferences.EnableSystemOps, "enable-system-operator", "", false, "DEPRECATED: use --security-enable-system-operator instead. Enable system operator to allow execution of external commands.")


@copilot no need to add enable-system-operator in this PR only to deprecate it. Also MarkPersistentFlagDeprecated doesn't exist anyway

Done in b3b4478. Removed the deprecated --enable-system-operator alias and the MarkPersistentFlagDeprecated call entirely. The flag is now only --security-enable-system-operator, consistent with --security-disable-env-ops and --security-disable-file-ops. All references in the operator warning message, test yqFlags, and docs are updated accordingly.

…le-system-operator consistently Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/286b95e9-b6d7-4ab8-b401-2d7a03853922 Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

mikefarah · 2026-04-06T08:42:39Z

@jandubois - I don't suppose you could run your deep review tool against this PR as well? That report was really neat..what are you using? Is this something you made?

jandubois · 2026-04-06T21:20:31Z

I don't suppose you could run your deep review tool against this PR as well?

I did run it: https://jandubois.github.io/yq/20260406-103045-pr-2640.html

There is also a Markdown version at https://jandubois.github.io/yq/20260406-103045-pr-2640.md if you want to copy some of it into a comment here, to feed back to Copilot. Be aware that the Markdown version includes the individual agent responses, so is quite verbose; you don't want to copy all of it. The HTML version is post-processed and has added code blocks and some other minor things.

Since this was another multi-pass review I now ran out of quota for Gemini for today. Unfortunately I only have a limited subscription for it and it looks like running 2 multi-pass reviews use up all of it. 😢

I can still run the tool, but have to run it with just Claude and Codex until 11pm...

That report was really neat..what are you using? Is this something you made?

Yes, it is my own tool running on "Zilicon Island", my off-leash sandbox (like Jurassic Park, except there are roaming AI agents instead of dinosaurs).

It has a lead agent preparing prompts for review agents using the top reasoning models, and then combining their findings. In a --thorough review it will run the review agents multiple times after adding the combined earlier findings to the prompt ("I already know about these; find me something else").

It takes a bit of time (you can see this latest review ran for 90 minutes wall time).

It also generates metrics and feedback that I use to continuously improve it.

You can see all reviews I ran against the yq repo at https://jandubois.github.io/yq/.

You can see that I ran the tool (single-pass only) against my oss-fuzz branches. It found issues in each of them, that I fixed before creating the PRs.

Those PRs were also all made with AI, but of course with some steering.

PS: Just realized that some of the code blocks have off-by-one errors in the line numbers and highlighted lines. So added a script to the tool to verify the code blocks, so the agent can check and fix it by themselves for future reviews. This is perpetual work-in-progress...

jandubois · 2026-04-07T07:28:01Z

Regarding I6. system operator subsumes env/file security restrictions I think it would be best not just to add the section to the docs, but to actually make --security-enable-system-operator mutually exclusive with either of --security-disable-env-ops or --security-disable-file-ops. It should throw an error when you try to use the former with either of the latter ones.

Initial plan

d181cf8

Copilot AI assigned Copilot and mikefarah Mar 27, 2026

Copilot started work on behalf of mikefarah March 27, 2026 10:04 View session

Add system(command; args) operator with --enable-system-operator flag

9a72c0f

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/8a11e9a0-10d2-4f2a-ae29-4e9d0bfc266f Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Add system command operator for yq transformations~~ Add system(command; args) operator (disabled by default) Mar 27, 2026

Copilot AI requested a review from mikefarah March 27, 2026 10:16

Copilot finished work on behalf of mikefarah March 27, 2026 10:17

mikefarah requested a review from Copilot March 27, 2026 11:08

Copilot started reviewing on behalf of mikefarah March 27, 2026 11:08 View session

Copilot AI reviewed Mar 27, 2026

View reviewed changes

Copilot started work on behalf of mikefarah March 28, 2026 01:58 View session

mikefarah and others added 2 commits March 28, 2026 12:59

Update pkg/yqlib/operator_system.go

6315cfe

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Evaluate system command/args per matched node using SingleReadonlyChi…

da611f7

…ldContext Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/dca841eb-3f63-4f23-adeb-556431560420 Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

Copilot finished work on behalf of mikefarah March 28, 2026 02:04

Copilot started work on behalf of mikefarah March 28, 2026 02:04 View session

Add yqFlags to expressionScenario for doc command snippets; fix syste…

884c2d8

…m op docs Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/3f8a5375-25fd-4428-a8e6-b630194c36b2 Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

Copilot finished work on behalf of mikefarah March 28, 2026 02:08

mikefarah requested a review from Copilot March 28, 2026 03:09

Copilot started reviewing on behalf of mikefarah March 28, 2026 03:09 View session

Copilot AI reviewed Mar 28, 2026

View reviewed changes

Copilot started work on behalf of mikefarah March 28, 2026 03:21 View session

mikefarah and others added 3 commits March 28, 2026 14:21

Update pkg/yqlib/doc/operators/headers/system-operators.md

2c8605f

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update pkg/yqlib/doc/operators/system-operators.md

5ea069a

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Validate command node type and handle multiple results with debug log

53abbba

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/928aabc5-ad71-41d8-94ab-403942e3f92d Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

Copilot finished work on behalf of mikefarah March 28, 2026 03:28

mikefarah marked this pull request as ready for review March 28, 2026 09:37

mikefarah requested a review from Copilot March 28, 2026 09:37

Copilot started reviewing on behalf of mikefarah March 28, 2026 09:37 View session

Copilot AI reviewed Mar 28, 2026

View reviewed changes

Apply suggestions from code review

e10e812

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

mikefarah reviewed Apr 6, 2026

View reviewed changes

Copilot started work on behalf of mikefarah April 6, 2026 08:33 View session

Remove deprecated --enable-system-operator alias; use --security-enab…

b3b4478

…le-system-operator consistently Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/286b95e9-b6d7-4ab8-b401-2d7a03853922 Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

Copilot finished work on behalf of mikefarah April 6, 2026 08:36

Copilot AI requested a review from mikefarah April 6, 2026 08:36

Apply suggestions from code review

6f94991

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Uh oh!

Conversation

Copilot AI commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Behaviour

Changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

mikefarah Mar 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI Mar 28, 2026

Choose a reason for hiding this comment

Uh oh!

mikefarah Mar 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Mar 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mikefarah Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

mikefarah commented Apr 6, 2026

Uh oh!

jandubois commented Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jandubois commented Apr 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

Copilot AI commented Mar 27, 2026 •

edited

Loading

jandubois commented Apr 6, 2026 •

edited

Loading